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Abstract 

Most common public key cryptosystems and public key exchange protocols presently 
in use, such as the RSA algorithm, Diffie-Hellman, and elliptic curve methods are num- 
ber theory based and hence depend on the structure of abelian groups. The strength 
of computing machinery has made these techniques theoretically susceptible to attack 
and hence recently there has been an active line of research to develop cryptosystems 
and key exchange protocols using noncommutative cryptographic platforms. This line 
of investigation has been given the broad title of noncommutative algebraic cryp- 
tography. This was initiated by two public key protocols that used the braid groups, 
one by Ko, Lee et.al.and one by Anshel, Anshel and Goldfeld. The study of these 
protocols and the group theory surrounding them has had a large effect on research 
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in infinite group theory. In this paper we survey these noncommutativc group based 
methods and discuss several ideas in abstract infinite group theory that have arisen 
from them. We then present a set of open problems. 
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1 Introduction and Nonabelian Group Based Cryptog- 
raphy 

Traditionally cryptography is the science and/or art of devising and implementing secret 
codes or cryptosystems. Cryptanalysis is the science and/or art of breaking cryptosys- 
tems while cryptology refers to the whole field of cryptography plus cryptanalysis. In most 
modern literature cryptography is used synonomously with cryptology. Presently there is 
an increasing need for secure cryptosystems due to the use of internet shopping, electronic 
financial transfers and so on. 

Most common public key cryptosystems and public key exchange protocols presently in use, 
such as the RSA algorithm, DifRe-Hellman, and elliptic curve methods are number theory 
based and hence theoretically depend on the structure of abelian groups. Althoiigh there 
have been no successful attacks on the standard protocols there is a feeling that the strength 
of computing machinery has made these techniques less secure. As a result of this there has 
been an active line of research to develop and analyze new cryptosystems and key exchange 
protocols based on noncommutativc cryptographic platforms. This line of investigation has 
been given the broad title of noncommutative algebraic cryptography (see [MSU]). 

Up to this point the main sources for noncommutative cryptographic platforms has been 
nonabclian groups. In cryptosystems based on these objects algebraic properties of the 
platforms are used prominently in both devising cryptosystems and in cryptanalysis. In 
particular the difficulty, in a complexity sense, of certain algorithmic problems in finitely 
presented groups, such as the conjugator search problem, has been crucial in encryption and 
decryption. 

The main sources for nonabelian groups are combinatorial group theory and linear group 
theory. Braid group cryptography (see [D]), where encryption is done within the classical 
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braid groups, is one prominent example. The one way functions in braid groiip systems are 
based on the difficulty of solving group theoretic decision problems such as the conjugacy 
problem and conjugator search problem. Although braid group cryptography had initial 
spectacular success, various potential attacks have been identified. Borovik, Myasnikov, 
Shpilrain [BMS] and others have studied the statistical aspects of these attacks and have 
identitifed what are termed black holes in the platform groups outside of which present 
cryptographic problems. Baumslag. Fine and Xu in [BFX] and [X] suggested potential 
cryptosystems using a combination of combinatorial group theory and linear groups and a 
general schema for these types of cryptosystems was given. In [BFX 2] a public key version of 
this schema using the classical modular group as a platform was presented. A cryptosystem 
using the the extended modular group (^) was developed by Yamamura ([Y]) but was 
subsequently shown to have loopholes ([BG], [S], [HGS]). In [BFX 2] attacks based on these 
loopholes were closed. 

The study and cryptanalysis of potential platform groups has had a strong positive effect on 
both group theory and complexity theory. Motivated in large part by cryptography, there 
has been tremendous interest in asymptotic group theory and generic properties (see 
sections 9 and 10). 

In this article we will present an overview of these combinatorial group theoretic methods. 
We will introduce free group cryptography and then the seminal Anshel-Anshel-Goldfeld 
and Ko-Lee Protocols. We will further discuss how to use combinatorial group theory in 
digital signatures and password verification. Then we will discuss potential platform groups 
and give a brief review of braid group cryptography. Next we introduce asymptotic density 
and its application to cryptanalysis and evaluation of cryptosystems. In particular we will 
look at some theoretical results concerning the generic free group property. Finally a list of 
open problems (in no way exhaustive) in this area will be presented. 

There are several recent books on this area. The most comprehensive is by Myasnikov, 
Shpilrain and Ushakov [MSU]. In this article we touch on several topics that are not men- 
tioned in that book, for example polycylic group cryptography, digital signature protocols 
and password authentication and the generic free group property. 

2 Basics of Public Key Cryptography 

In this section we describe the standard terminology used in cryptography and then intro- 
duce the two most common public key methods, the Diffie-Hellman protocol and the RSA 
protocol. 

In general both the plaintext message (uncoded message) and the ciphertext message 

( coded message) are written in some A^-lctter alphabet which is usually the same for both 
plaintext and code. The method of coding or the encoding algorithm is then a transformation 
of the A/'-letters. The most common way to perform this transformation is to consider the 
TV letters as TV integers modulo TV and then perform a number theoretical fiuiction on 
them. Therefore most encoding algorithms use modular arithmetic and hence cryptography 
is closely tied to number theory. 
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Modern cryptography is usually separated into classical or symmetric key cryptography 
and public key cryptography. In the former, both the encoding and decoding algorithms 
are supposedly known only to the sender and receiver, usually referred to as Bob and Alice. 
In the latter, the encryption method is public knowledge but only the receiver knows how 

to decode. 

The process of putting the plaintext message into code is called enciphering or encryption 
while the reverse process is called deciphering or decryption. Encryption algorithms 
partition the plaintext and ciphertext message into message units. These are single letters 
or pairs of letters or more generally /c- vectors of letters. The transformations are done on 
these message units and the encryption algorithm is a mapping from the set of plaintext 
message units to the set of ciphertext message units. Putting this into a mathematical 
formulation we let 

V = {set of all plaintext message units } and 
C = { set of all ciphertext message units }. 
The encryption algorithm is then the application of an invertible function 

f:V^C. 

The function / is the encryption map. The inverse 

/-I :Ch->P 

is the decryption or deciphering map. The triple {'P,C, /}, consisting of a set of plain- 
text message units, a set of cipertext message units and an encryption map is called a 
cryptosystem. 

Breaking a code is called cryptanalysis. An attempt to break a code is called an attack. 

Most cryptanalysis starts with a statistical frequency analysis of the plaintext language 
used. Cryptanalysis depends also on a knowledge of the form of the code, that is, the type 
of cryptosystem used (see [K] or [MSU]). 

Most classical cryptosystems are number theoretically derived crytosystems. In applying 
a cryptosystem to an A'' letter alphabet we consider the letters as integers mod N. The 
encryption algorithms then apply number theoretic functions and use modular arithmetic 
on these integers. 

We usually do not use a single letter at a time but rather a sequence of k letters. The k 
letters are then a message unit. An encryption algorithm is then a function 

Presently there are many instances where secure information must be sent over open com- 
munication lines. These include for example banking and financial transactions, purchasing 
items via credit cards over the internet and similar things. This led to the development of 
public key cryptography. Roughly, in classical cryptography only the sender and receiver 
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know the encoding and decoding methods. Further it is a feature of such cryptosystems 
that if the encrypting method is known the decrypting can be carried out. 

In public key cryptography the encryption method is pubhc knowledge but only the 
receiver knows how to decode. More precisely in a classical cryptosystem once the encrypting 

algorithm is known the decryption algorithm can be implemented in approximately the same 
order of magnitude of time. In a public the decryption algorithm is much more difficult to 
implement. This difficulty depends on the type of computing machinery used ( much as 
primality testing) and as computers get more powerful, new and more secure pulic key 

cryptosystems become necessary. 

The basic idea in a public key cryptosystem is to have a one-way function. That is a 
function which is easy to implement but very hard to invert. Hence it becomes simple to 
encrypt a message but very hard, unless you know the inverse, to decrypt. 

The standard model for a public key cryptosystem is the following. Alice wants to send 
a message to Bob. The encrypting map /a for Alice is public knowledge as well as the 
encrypting map fs for Bob. On the other hand the decryption algorithms f^^ and fg^ are 
secret and known only to Alice and Bob respectively. Let P be the message Alice wants to 
send to Bob. She sends fEfX^iV)- To decode Bob applies first fg^, which only he knows. 
This gives him = fX^CP)- He then looks up Ja which is publically available 

and applies this jAifA^CP)) = V to obtain the message. 

Alice sends /b/^^('P) rather than just /b(^) for authentication, that is being certain 
from Bob's point of view that the message really came from Alice. Suppose V is Alice's 
verification; signature, social security number etc.. If Bob receives fsiP) it could be sent by 
anyone since Jb is public. On the other hand since only Alice supposedly knows /^^ getting 
a reasonable message from /aI/b^/b/a^I^)) would verify that it is from Alice. Applying 
alone should result in nonsense. 

Getting a reasonable one way hmction can be a formidable task. The most widely used (at 
present) public key systems are based on the difficulty of inverting certain number theoretic 
functions. The first real public key protocol was developed in 1976 by Diffie and Hellman 
using the difficulty of the discrete log problem. 

In modular arithmetic it is easy to raise an clement to a power but difficult to determine, 
given an element, if it is a power of another element. Specifically if G is a finite group, such 
as the cyclic multiplicative group of Zp where p is a prime, and h = g'^ for some k then 
the discrete log of h to the base g is any integer t with h = g*. The rough form of the 

Difhe-Hclman public key system is as follows. 

Bob and Alice will use a classical cryptosystem based on a key k with 1 < A: < g — 1 where q 
is a prime. It is the key k that Alice must send to Bob. Let be a multiplicative generator 
of Z*. Alice chooses an a S with 1 < a < g — 1. She makes public g"-. Bob chooses a 
b £ Zg and makes public g''. The secret key is g"''^. Both Bob and Alice, but presumably 
noone else, can discover this key. Alice knows her secret power a and the value g'' is public 
from Bob. Hence she can compute the key g""^ = {g^Y ■ The analogous situation holds 
for Bob. An attacker however only knows g, g°' and g''. Unless the attacker can solve the 
discrete log problem, that is finding a or 6 the key exchange is secure. 
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Notice that this depends upon 

As we will see, the Ko-Lee protocol exactly mimics the Diffie-Hellman protocol within a 
nonabelian group by interpreting powers as conjugation. 

In 1997 it became known that the ideas of public key cryptography were developed by 
British Intelligience Services prior to Diffie and Hellman. 

In 1977 Rivest, Adelman and Shamir developed the RSA Algorithm which is presently 
the most widely used public key cryptosystems. It is based on the difficulty of factoring 
large integers and in particular on the fact that it is easier to test for primality than to 
factor. In basic outline at the simplest level it works as follows. 

Alice chooses two large primes pa,(1a and an integer ca relatively prime to ^{paQa) = 
(pa — 1)('Za — !)• It is assumed that these integers are chosen randomly to minimize attack. 
The primes she chooses should be quite large. Originally RSA used primes of approximately 
100 decimal digits, but as computing and attack have become more sophisticated, larger 
primes have had to be utilized. Once Alice has obtained pa,CIa,£a she lets ua = PaIa 
and computes cIa , the multiplicative inverse of ca modulo ^(ua)- That is (Ia satisfies 
CAdA = 1 niod {pa — ^){qA — !)• She makes public the enciphering key Ka = {nA,eA) and 
the encryption algorithm known to all is 

fA(V) = mod UA 

where V g Z„^ is a message unit. 

It can be shown that if (e^i, [pA — 1)('?a ^ 1)) = 1 and e^d^ = 1 mod {pa — — 1) then 

•pe^idA = -p jnod UA Therefore the decryption algorithm is 

/-1(C) =C''-* mod UA. 

Notice then that /^^(/aCP)) = V^''-^ = V mod ha so it is the inverse. 

Now Bob makes the same type of choices to obtain Pb,<Ib,gb- He lets Ub = PbQb and 
makes public his key Kb = (ns, &b)- 

If Alice wants to send a message to Bob that can be authenticated to be from Alice she 
sends An attack then requires factoring ua or ub which is much more difficult 

than obtaining the primes pA,qA,PB,qB- 

Again notice the use of commutativity. 

There have been many extensions and enhancements of these basic public key protocols. 
Elliptic curve cryptography uses the discrete log problem within the group of an elliptic 
curve. This group is a finite abelian group and has certain advantages over the cyclic groups 
used in the standard DifBe-Hellman protocol. The book by Koblitz [K] has a thorough 
description of elliptic curve methods. 

The El-Gamal cryptosystem is a technique to use the DifHe-Hellman key exchange method 
to do encryption. The method works as follows. Suppose that Bob and Alice want to 
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communicate openly. They have exchanged a secret key k that supposedly only they know. 
Let fk be an encryption function or encryption algorithm based on the key k. Alice wants 
to send the message m to Bob and m is given as a binary bit string. Alice sends to Bob 

fk{m) © k 

where fc is a bit string for the key k and ® is addition modulo 2. 

Bob knows the key k and hence can compute it as a bibary string. He now computes 

fk{m)®k®k 

Since addition modulo 2 has order 2 we have 

fk{m)®k®k = fk{m). 

Bob now applies the decryption algorithm f^^ to decode the message. In practice a hash 
function is usually applied to k (see [B]). 

3 The Fundamentals of Free Group Cryptography 

The extension of all these ideas to noncommutative platforms is the subject of noncom- 
mutative algebraic cryptography. This involves the following ideas, 

(1) General Algebraic Techniques for Developing Cryptosystems 

(2) Potential Algebraic Platforms (Specific Groups, Rings, Etc.) for implementing the 

Techniques 

(3) Cryptanalysis and Security Analysis of the Resulting Systems 

The main source for noncommutative platforms are nonabelian groups and the main method 
for handling nonabelian groups in cryptography is combinatorial group theory. This refers to 
the branch of group theory that studies groups by using group presentations, that is sets of 
generators and relations between them. The basic idea in using combinatorial group theory 
for cryptography is that elements of groups can be expressed as words in some alphabet. If 
there is an easy method to rewrite group elements in terms of these words and further the 
technique used in this rewriting process can be supplied by a secret key then a cryptosystem 
can be created. The simplest example is perhaps a free group cryptosystem. This can 
be described in the following manner. 

Consider a free group F on free generators xi,...,Xr- Then each element g in F has a 
unique expression as a word W{xi, x^). Let Wi, Wk with Wi = Wi{xi, Xr) be a set 
of words in the generators xi, of the free group F. At the most basic level, to construct 
a cryptosystem. suppose that we have a plaintext alphabet A. For example suppose that 
A = {a,b,...} are the symbols needed to construct meaningful messages in English. To 
encrypt, use a substitution ciphertext 

A^{Wi,...,Wk}. 
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That is 



a M> VFi, 6 W2, ■ 



Then given an word W{a,b, ...) m the plaintext alphabet form the free group word W2, ■■■■)■ 

This represents an element g in F. Send out g as the secret message. 

In order to implement this scheme we need a concrete representation of g and then for 
decryption a way to rewrite g back in terms of Wi, W/-. This concrete representation is 
the idea behind homomorphic cryptosystems. 

The decryption algorithm in a free group cryptosystem then depends on the Reidemeister- 
Schreier rewriting process. This is a method to rewrite elements of a subgroup of a free 

group in terms of the generators of that subgroup. We refer to [MKS] or [GB 1] for a complete 
description of the technique. Roughly it works as follows. Assume that Wi, Wk are free 
generators for some subgroup H oia, free group F on {xi, ...,Xn}- Each Wj is then a reduced 
word in the generators {xi, a;„}. A Schreier transversal for H is a set {hi, ht, ...} of 
(left) coset representatives for _ff in of a special form (see [MKS]). Any subgroup of a free 
group has a Schreier transversal. The Reidemeister-Schreier process allows one to construct a 
set of generators Wi , Wk for H by using a Schreier transversal. Further given the Schreier 
transversal from which the set of generators for H was constructed, the Reidemeister- 
Schreier Rewriting Process allows us to algorithmically rewrite an element of H. Given 
such an element expressed as a word W = W{xi , Xr) in the generators of F this algorithm 
rewrites as a word W*(M^i, Wk) in the generators of H. 

The knowledge of a Schreier transversal and the use of Reidemeister-Schreier rewriting 
facilitates the decoding process in the free group case but is not essential. Given a known 
set of generators for a subgroup the Stallings Folding Method to develop a subgroup graph 
can also be utilized to rewrite in terms of the given generators. The paper by Kapovich and 
Myasnikov [KM] is now a standard reference for this method in free groups. At present there 
is an ongoing study of the complexity of Reidemeister-Schreier being done by Baumslag, 
Brukhov, Fine and Troeger. 

Pure free group cryptosystems are subject to various attacks and can be broken easily. 
However a public key free group cryptosystem using a free group representation in the 
Modular group was developed by Baumslag, Fine and Xu [BFX 1 2]. The most successful 
attacks on free group cyrpotsystems are called length based attacks. Here an attacker 
multiplies a word in ciphertext by a generator to get a shorter word which could possibly 
be decoded. 

Baumslag, Fine and Xu in [BFX 1] described the following general encryption scheme using 
free group cryptography. A further enhancement was discussed in the paper [BFX 2]. 

We start with a finitely presented group 

G=<X\R> 

where X = {xi, a;„} and a faithful representation 
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G can be any one of several different kinds of objects - linear group, permutation group, 

power series ring etc. 

We assume that there is an algorithm to re-express an element of p(G) in G in terms of 
the generators of G. That is if 5 = W{xi,...,Xn, •••) G G where is a word in the these 
generators and we are given p{g) G G we can algorithmically find g and its expression as 

the word W{xi, x„). 

Once we have G we assume that we have two free subgroups K, H with 

HcK CG. 

We assume that we have fixed Schreier transversals for K inG and for H in K both of which 
are held in secret by the communicating parties Bob and Alice. Now based on the fixed 
Schreier transversals we have sets of Schreier generators constructed from the Reidemeister- 
Schreier process for K and for H. 

ki,...km,— ioT K 

and 

hi,...,ht,... fori?. 

Notice that the generators for K will be given as words in xi, ...,.x„, the generators of G 
while the generators for H will be given as words in the generators ki,k2,..-. for K. We 
note further that H and K may coincide and that H and K need not in general be free but 
only have a unique set of normal forms so that the representation of an element in terms of 

the given Schreier generators is unique. 

We will encode within H, or more precisely within p{H). We assume that the number of 
generators for H is larger than the set of characters within our plaintext alphabet. Let 
A = {a, b, c...} be our plaintext alphabet. At the simplest level we choose a starting point 
i, within the generators of H, and enclodc 

a hi,b 1-^ hi+i, .... etc. 

Suppose that Bob wants to communicate the message W(a,b,c...) to Alice whore W is a 
word in the plaintext alphabet. Recall that both Bob and Alice know the various Schreier 
transversals which are kept secret between them. Bob then encodes W{hi, hi+i...) and 
computes in G the element W {p{hi) , p{hi+i) , ..) which he sends to Alice. This is sent as a 
matrix if G is a linear group or as a permutation if G is a permutation group and so on. 

Alice uses the algorithm for G relative to G to rewrite W{p{hi), p{hi+i), ..) as a word 
W*{xi, ...Xn) in the generators of G. She then uses the Schreier transversal for K in G 
to rewrite, using the Reidemeister- Schreier process, W* as a word W**{ki, ...,ks..) in the 
generators of K. Since K is free or has unique normal forms this expression for the element 
of K is unique. Once she has the word written in the generators of K she uses the transver- 
sal for H in K to rewrite again, using the Reidemeister-Schrcicr process, in terms of the 
generators for H. She then has a word W***(/i,, /ij+i, ...) and using hi a, /ij+i b,... 
decodes the message. 
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In actual implementation an additional random noise factor is added. 

In [FBX 1,2] an inplcmentation of this process was presented that used for the base group G 
the classical modular group M = PSL{2,Z). Further it was a polyalphabetic cipher which 
was secure. 

The system in the modular group M was presented as follows. A list of finitely generated free 

subgroups Hi, H„i of M is pubhc and presented by their systems of generators (presented 
as matrices). In a full practical implementation it is assumed that m is large. For each Hi 
we have a Schreier transversal 

and a corresponding ordered set of generators 

constructed from the Schreier transversal by the Reidemeister-Schreier process. It is as- 
sumed that each m{i) >> I where I is the size of the plaintext alphabet, that is each 
subgroup has many more generators than the size of the plaintext alphabet. Although Bob 
and Alice know these subgroups in terms of free group generators, what is made public are 
generating systems given in terms of matrices. 

The subgroups on this list and their corresponding Schreier transversals can be chosen in a 
variety of ways. For example the commutator subgroup of the Modular group is free of rank 
2 and some of the subgroups Hi can be determined from homomorphisms of this subgroup 
onto a set of finite groups. 

Suppose that Bob wants to send a message to Alice. Bob first chooses three integers (m, q, t) 

where 

m = choice of the subgroup Hm 

q = starting point among the generators of H^ 

for the substitution of the plaintext alphabet 

t = size of the message unit . 

We clarify the meanings of q and t. Once Bob chooses m, to further clarify the meaning of 
q, he makes the substitution 

a Wm,q,b>-^ Wm,q+1, 

Again the assumption is that m{i) » I so that starting almost anywhere in the sequence 
of generators of Hm will allow this substitution. The message unit size t is the number of 
coded letters that Bob will place into each coded integral matrix. 

Once Bob has made the choices (m, q, t) he takes his plaintext message W{a, b, ...) and groups 
blocks of t letters. He then makes the given substitution above to form the corresponding 
matrices in the Modular group; 
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Wc now introduce a random noise factor. After forming Ti, ...,Ts Bob then multiplies on 
the right each Tj by a random matrix in M, say Rt^ ( different for each Ti). The only 
restriction on this random matrix Rt^ is that there is no free cancellation in forming the 
product TiRxi. This can be easily checked and ensures that the freely reduced form for 
Tji?Ti is just the concatenation of the expressions for Tj and Rt^- Next he sends Alice the 
integral key (m, q, t) by some public key method (RSA, Anshel-Goldfeld etc.). He then sends 
the message as s random matrices 

TiRti , T2RT2 J •••> TgRxs . 

Hence what is actually being sent out are not elements of the chosen subgroup Hm but rather 
elements of random right cosets of Hm in AI . The purpose of sending coset elements is two- 
fold. The first is to hinder any geometric attack by masking the subgroup. The second 
is that it makes the resulting words in the Modular Group generators longer, effectively 
hindering a brute force attack. 

To decode the message Alice first uses public key decryption to obtain the integral keys 
{m,q,t). She then knows the subgroup Hm, the ciphertext substitution from the gener- 
ators of Hm and how many letters t each matrix encodes. She next uses the algorithms 
described above to express each TiRt^ in terms of the generators of M. She has knowl- 
edge of the Schreier transversal, which is held secretly by Bob and Alice, so now uses the 
Reidemeister-Schreier rewriting process to start expressing this freely reduced word in terms 
of the generators of Hm- The Reidemeister-Schreier rewriting is done letter by letter from 
left to right (see [MKS]). Hence when she reaches t of the free generators she stops. Notice 
that the string that she is rewriting is longer than what she needs to rewrite in order to 
decode as a result of the random matrix Rj-. . This is due to the fact that she is actually 
rewriting not an element of the subgroup but an element in a right coset. This presents a 
further difficulty to an attacker. Since these are random right cosets it makes it difficult to 
pick up statistical patterns in the generators even if more than one message is intercepted. 
In practice the subgroups should be changed with each message. 

The initial key (m, q, t) is changed frequently. Hence as mentioned above this method 
becomes a type of polyalphabetic cipher. Polyalphabetic ciphers have historically been very 
difficult to decode. 

A further variation of this method using the Magnus representation in a formal power series 
ring in noncommuting variables over a field was described in [BBFR]. 

4 Public Key Exchange Using Nonabelian Groups 

Among the first attempts to use nonabelian groups in cryptography were the schemes of 
Anshel-Anshel-Goldfeld[AAG] and Ko-Lee et.al.[KoL]. Both sets of authors, at about the 
same time, proposed using nonabelian groups and combinatorial group theory for public 
key exchange. The security of these systems depended on the difficulty of solving certain 
"hard" group theoretic problems. 
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The methods of both Anshel-Anshel-Goldfeld and Ko-Lee can be considered as group the- 
oretic analogs of the number theory based Difhe-Hellman method. The basic underlying 
idea is the following. If G is a group and g,h G G we let g'^ denote the conjugate of g by 
h, that is g^ = h^^gh. The simple observation is that this behaves like ordinary exponen- 
tiation in that (5''^)''^ = g^^^'^ . From this straightforward idea one can exactly mimic the 
Diffie-Hellman protocol within a nonabelian group. 

Both the Anshel-Anshel-Goldfeld protocol and the Ko-Lee protocol start with a platform 

group G given by a group presentation. A major assumption in both protocols is that the 
elements of G have nice unique normal forms that are easy to compute for given group 
elements. However it is further assumed that given normal forms for x,y £ G, the normal 
form for the product xy, does not reveal x or y. 

We describe the Anshel-Anshel-Goldfeld public key exchange protocol first. Let G be the 
platform group given by a finite prsentation and with the assumptions on normal forms as 
described above. 

Alice and Bob want to communicate a shared secret. First, Alice and Bob choose random 
finitely generated subgroups of G by giving a set of generators for each. 

A = {ai, an}, B = {hi, 

and make them public. The subgroup A is Alice's subgroup while the subgroup B is Bob's 
subgroup. 

Alice chooses a secret group word a = W{ai^ ■■■,an) in her subgroup while Bob chooses a 
secret group word b = V{bi, ...,bm) in his subgroup. For an element <? G G we let NF{g) 
denote the normal form for g. Alice knows her secret word a and knows the generators bi 
of Bob's subgroup. She makes public the normal forms of the conjugates 

NF{b^),i = l,...,m. 

Bob knows his secret word b and the geerators ai of Alice's subgroup and makes public the 
normal forms of the conjugates 

NF{a''j),j = l,...,n). 
The common shared secret is the commutator 

[a, b] = a'H-^ab = a'^a^ = {b^y^b 

Notice that Alice knows since she knows a in terms of generators of her subgroup and 
she knows the conjugates by b since Bob has made the conjugates of the generators of A by 
b public. Since Alice knows she knows [a, b] = a~^a^. 

In an analogous manner Bob knows [a, b] = {b°-)~^b. An attacker would have to know the 
corresponding conjugator, that is the clement that conjugates each of the generators. Given 
elements g,h in a, group G where it is known that g'' = k~^gk = h the conjugator search 
problem is to determine the conjugator k. It is known that this problem is undecidable in 
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general, that is there are groups where the conjugator cannot be determined algorithmicahy. 
On the other hand there are groups where the conjugator search problem is solvable but 
"difficult", that is the complexity of solving the conjugator search problem is hard. Such 
groups become the ideal platform groups for the Anshel-Anshel-Goldfeld protocol. 

The security in this system is then in the difficulty of the conjugator search problem. 
Anshel, Anshel, Goldfeld suggested the Braid Groups as potential platforms and use for 
example Bgo with 12 or more generators in the subgroups. Their suggestion and that of 
Ko and Lee led to development of braid group cryptography. There have been various 
attacks on the Braid group system. However some have been handled by changing the 
parameters. In general the ideas remain valid despite the attacks. We will discuss this 
further in section 7. 

Ko, Lee et. al. [KoL] developed a similar system that is a direct translation of the Diffie- 
Hellman protocol to a nonabelian group theoretic setting. Its security is based on the 
difficulty of the conjugacy problem. We again assume that the platform group has nice 
unique normal forms that are easy to compute for a given group element but hard to recover 
the group element. Recall again that means the conjugate of g by /i 

In the Ko-Lee protocol, Alice and Bob choose commuting subgroups A and B of the platform 
group G. A is Alice's subgroup while Bob's subgroup is B and these are secret. Now they 
completely mimic the classical Diffie-Hellman technique. There is a public element g € G, 
Alice chooses a random secret element a G A and makes public 5". Bob chooses a random 
secret element b E B and makes public g''. The secret shared key is g'^^. Notice that ab — ba 
since the subgroups commute. It follows then that (5")'' = g""^ = g^°' = [g^Y J^^t as if 
these were exponents. Hence both Bob and Alice can determine the common secret. The 
difficulty is in the difficulty of the conjugacy problem. 

The conjugacy problem for a group G, or more precisely for a group presentation for G, 
is given g,h € G to determine algorithmicahy if they are conjugates. As with the conjugator 
search problem it is known that the conjugacy is undecidable in general but there are groups 
where it is but hard. These groups then become the target platform groups for the Ko-Lee 
protocol. As with the Anshel-Anshel-Goldfeld protocol, Ko and Lee suggest the use of the 
Braid groups. 

The conjugacy problem and the conjugator search problem are only two of the group theo- 
retic search and decision problems that have been employed to construct one way functions 
in a cryptogrpahic setting. We recall several other important such problems and then ref- 
erence their use in encryption and public key exchange . 

Definition 4.1 (Word Problem). Given a finitely presented group G, does there exist an 
algorithm to decide whether or not a word in the generators is the trivial word? 

Definition 4.2 (Decision Conjugacy Problem). Given a group G with a finite presentation, 
does there exist an algorithm to decide whether or not an arbitrary pair of words u and v in 
the generators of G are conjugate? That is, is there an x € G such that x~^ux = v? 

Definition 4.3 (Decomposition Problem). Let G be a finitely presented group with sub- 
groups A,B < G. Given two elements u and v of G, is there an algorithm to find two 
elements a & A and b G B such that aub = v? 
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Definition 4.4 (Simultaneous Search Conjugacy problem). Let G be a finitely presented 
group. Given ui, • • • , Uk, vi, ■ ■ ■ ,Vk € G with x~^UiX = Vi for each i € {1, 2, • • • , k}, is there 
an algorithm to find z G G satisfying z~^UiZ = Vi for each i G {1,2, ...,k}? 

We now reference some uses for these problems. Wagner, Birget, Magliaveras and Sramka 
developed key exchange protocols based on the word problem. Kurt developed a protocol 
based on the decomposition problem. Shpilrain and Ushakov developed a protocol based 
on the twisted conjugacy problem while Shpilrain and Zapata developed several encryption 
protocols based on various decision problems. Complete descriptions of these protocols 
can be found in [MSU] . More recently Anshcl and Kahrobaei developed a noncommutative 
analog of the Cramcr-Shoup key exchange method [AK]. 

We close this section by describing a noncommutative analog of the El Gamal public key 
exchange system based on the search conjugacy problem. It was proposed by Kahrobaei 
and Khan [KKh]. As with the Ko-Lee and Anshel-Anshel-Goldfeld protocols we start with 
a finitely presented platform group G given by a group presentation. As before the major 
assumptions are that the elements of G have nice unique normal forms that are easy to 
compute for given group elements. However it is further assumed that given normal forms 
for x,y G G the normal form for the product xy does not reveal x or y. Further G contains 
two commuting finitely generated proper subgroups S and T. The cryptographic goal is for 
Alice and Bob to establish a session key over an unsecured network. 

Bob chooses a secret element s E S and an arbitrary element b G G. Bob publishes b and 
= 6". Suppose Alice wants to send x G G as a session key to Bob. Then, 

1. Alice chooses a random t G T and sends E = a;^'^*) to Bob along with the header 
h = 5*. 

2. Bob then calculates (6*)^ = (6")* = c*. 

3. Now, Bob may calculate E' = (c*)~^, allowing him to decrypt the session key since 

The feasibility of this scheme relies on the assumption that products and inverses in G 
can be computed efficiently. Determining Bob's private key s entails solving the search 
conjugacy problem for G. That is given c, b, and c — b^ , determine s. Hence, the security 
of this scheme is based on the assumption that there is no practical algorithm for solving 
the search conjugacy problem for G. 

A second El Gamal analog based on the search power conjugay problem was also proposed 
by Kahrobaei and Khan (see [KKh]). 

5 The Shamir Three Pass and Key Transport Protocols 

A key treinsport protocol is a method that allows the sending of a key (telling for example 
what encryption system to use) from one user to another over a public airway. A group 
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theoretic key transport protocol based on the Difhc-Henman scheme can be developed in 
the following manner. Suppose that we have a finitely presented group G with the same 
assumptions made as in the Anshel-Anhsel-Goldfeld and Ko-Lee protocols. That is, G is 
given by a presentation and the elements of G have nice; normal forms. Further it is assumed 
that G has two large subgroups Ai,A2 that commute elementwise. Alternatively we could 
use one large abelian subgroup A of G. The meaning of large is of course hazy but here 
means that within G it is difficult to determine when an arbitrary element is in A (or Ai, A2) 
and further A (or Ai,A2) is large enough so that random choices can be made from them. 

Now suppose that Bob wants to communicate with Alice via an open airway. The secret key 
telling them which encryption system to use is encoded within the finitely generated group 
G with the properties given above. The two subgroups Ai,A2 which commute elementwise 
are kept secret by Bob and Alice. Ai is the subgroup for Bob and A2 the subgroup for Alice. 
Bob wants to send the key W € G to Alice. He chooses two random elements Bi,B2 S Ai 
and sends Alice the message ( in encrypted form) BiWB2- Alice now chooses two random 
elements Ci,G2 € A2 and sends C1B1WB2C2 back to Bob. These messages appear in the 
representation of G and hence for example as matrices or as reduced words in the generators 
so they don't appear as solely concatenation of letters. Since Ai commutes elementwise with 
A2 we have 

C1B1WB2C2 = B1C1WC2B2. 

Further since Bob knows his chosen elements Bi and B2 he can multiply by their inverses to 
obtain C1WC2 which he then sends back to Alice. Since Alice knows her chosen elements 
Ci, C2 she can multiply by their inverses to obtain the key W. It is assumed that for each 
message Bob and Alice would choose different pairs of random elements from either Ai or 
A2. This method is known as a Shamir Three- Pass which was introduced by Shamir for 
general algebraic objects. 

Notice that although this is roughly based on the Diffic-HcUman method it is not symmetric 
in the communicating parties. In the present scheme the secret key is completely determined 
by Bob, who then communicates it to Alice. The scheme then falls into the class of key 
transport protocols rather than public key exchange protocols. Key transport protocols are 
in most cases designed assuming that an underlying encryption system (and usually also a 
signature verification system) is in place. The security of the key transport protocol will rely 
on the security of these auxiliary schemes. In the group theoretic proposal the encryption 
scheme is suggested to be done within the same group as the key transport protocol alhtough 
this is not essential. In the group theortic key transport protocol an attacker has knowledge 
of the overall group G and a view of encrypted messages. The security lies in the difficulty 
of determining the elementwise commuting subgroups Ai,A2, which are kept secret by Bob 
and Alice, and in the security of the actual encryption scheme. 

A group G is a candidate platform group for this type of key transport protocol if it has either 
a nice finite presentation G =< X,R> with workable normal forms and has either a large 
abelian subgroup A or two large subgroups Ai,^2 that commute elementwise. Although the 
word large here is ambiguous we mean large enough so that random choices can be made from 
them. In particular, for example, cyclic subgroups are inappropriate. There also should be 
some tie between the group used for the key exchange and the encryption method although 
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this is not essential. The standard braid groups, that wih be described in the section 7 have 
several possibilitiess for normal forms and have large commuting subgroups. Hence they 
are excellent candidates for this method. In [BCFRX] several additional potential platform 
groups were suggested. These include the full aiitomorphism group of a finitely generated 
free group, the matrix group 5'L(4, Z) and the surface braid groups. Shpilrain and Ushakov 
[SU] used this method employing Thompson's group F as a platform. A length based attack 
on their system was attempted by Tsaban (see [MSU]). Further work on this method in the 
surface braid groups was done by Camps [C]. 

6 Digital Signatures, Authentication and Password Se- 
curity 

Authentication is the process of determining that a message, supposedly from a given 

person, both does come from that person and has not been tampered with. Authentication 
plays a major role in transmitting encrypted messages. Often this takes the form of a digital 
signature. A signature scheme provides a way for each user to sign messages so that the 
signatures can later be verified by anyone else. More specifically, each user can create a 
matched pair of private and public signature for the message (using the signer's public key). 
The verifier can convince himself that the message contents have not been altered since the 
message was signed. Also the signer cannot later repudiate having signed the message, since 
no one but the signer possesses his private key. By analogy with the paper world, where 
one might sign a letter and seal it in an envelope, one can sign an electronic message using 
one's private key, and then seal the result by encrypting it with the recipient's public key. 
The recipient can perform the inverse operations of opening the letter and verifying the 
signature to electronic mail are quite widespread today already (see [GoB]) 

A digital signature scheme within the public key framework, is defined as a triple of algo- 
rithms {A, a, V) such that 

• Key generation algorithm A is a probabilistic, polynomial-time algorithm which on 

input a security parameter 1*^, produces pairs {P, S) where P is called a public key 
and S a secret key. (We use the notation (P, S) G G{l'^) to indicate that the pair 
(P, S) is produced by the algorithm A.) 

• Signing algorithm a is a probabilistic polynomial time algorithm which is given a 
security parameter 1*^, a secret key S in range ^(l'^), and a message m G {0,1}*' 
and produces as output string s which we call the signature of m. (We use notation 
s e (t(1'^, S, m) if the signing algorithm is probabilistic and s = cr(l*', S, m) otherwise. 
As a shorthand when the context is clear, the secret key may be omitted and we will 
write s G a{S, m) to mean that s is the signature of message m.) 

• Verification algorithm y is a probabilistic polynomial time algorithm which given 
a public key P, a digital signature s, and a message to, returns 1 (i.e "true") or 
(i.e "false") to indicate whether or not the signature is valid. We require that 
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V{P, s, m) = 1 if s G (T(m) and otherwise. (We may omit the public key and 
abbreviate V{P, s, m) as V{s, m) to indicate verifying signature s of message m when 
the context is clear.) 

• The final characteristic of a digital signature system is its security against a proba- 
bilistic polynomial time forger. We delay this definition until later. 

We present a digital signature procedure based on nonabelian groups developed by Ko, Lee 
et al(see [KCCL]. Here is the scheme: 

Let G be a non-abelian group in which the search conjugacy problem is infeasible and the 
decision conjugacy problem is solvable. Let h : {0, 1}* — G be a hash function. 

Key Generation: Alice wants to sign and send a message, m, to Bob. Alice begins by 
choosing two conjugate elements u,v G G with conjugator a. The conjugate pair {u, v) is 
public information while the conjugator a is Alice's secret key. 

Signature Generation: Alice chooses arbitrary b G G, and computes a = and 
y = h{ma). Then a signature a on the message m is the triple (a, /3, 7) where p = 
and 7 = t/" She sends this to Bob for verification and acceptance. 

Verification: Upon receiving the signature, Bob checks whether or not the following hold: 

1. 3ci e G such that u = cf^. 

2. 3c2, C3 e G such that 7 = ^"'^ and y = ^''^. 

3. 3c4 e G such that uy = (aP)"*. 

4. 3c5 e G such that vy = (a'jY^ . 

Bob accepts the signature if and only if 1-4 hold. 

The security of this scheme lies in the assumption that given a pair of conjugate elements 
u,v G G finding elements a, (3, 7 such that 1-4 above hold is infeasible. If the conjugator 
a could be found then, then (a,/?, 7) = {u'',y'',y°- '') satisfy properties 1-4 for any b G G. 
Hence, the conjugacy search problem need be infeasible. 

We mention that there is digital signature scheme proposed by Anjaneyulu et. al [AVRR] 
that uses a noncommutative platform but outside of group theory. In this proposal the 
basic cryptographic platform is a commutative division semiring and uses what is teremed 
the polynomial syemmtrical decompisiton problem as the one way function. The reader can 

refer to the paper [AVVR] for details. 

Closely related to digital signatures is the problem of secure password verification. With 
the increased use of bank cards and internet credit card transactions there is at present 
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more than ever a need for secure password identification. For many online purchases this is 
being carried out by a challenge response system (see [W]) accompanying the password. 
In the simplest systems this takes the form of secondary password questions such as the 
user's mother's maiden name or place of birth. There are inherent difficulties with these 
types of challenge response systems. First of all there is the trivial problem of the users 
remembering their responses. More critical is the problem that this type of information 
for many people is readily available and easily found or guessed by would-be attackers or 
eavesdroppers. Challenge response systems are also subject to middleman attacks and replay 
attacks (see [CRW]. There have been several attempts to alleviate these problems, including 
zero- knowledge password proofs and challenged responses somewhat based on RSA as well as 
timed out responses (see CRAM-MD5, Password Authenticated Key Agreement, [CRRSA] 
and [W]). 

In [BBFT] an alternative method for challenge response password verification using combi- 
natorial group theory was developed which is provably secure against cipher-text only, man 
in the middle and replay attacks. In particular this method depends upon the difficulty of 
solving the word problem within a given finitely presented group without knowing the pre- 
sentation and the difficulty of solving systems of equations within free groups. This latter 
problem has been proved to be NP-hard. The method uses the group randomizer system 
which is a computer program that is a subset of MAGNUS a much larger computer algebra 
system designed to handle algorithm problems in combinatorial group theory, MAGNUS 
was developed at CAISS, the Center for Algorithms and Interactive Scientific Software, a 
research laboratory housed at City College of the City University of New York and under 
the direction of the first author. The group randomizer system can be placed on a simple 
hand held computer device presently under development at CAISS. The system can also be 
used from computer to computer (see [BBFT]). 

These group theoretic techniques have several major advantages over other challenge re- 
sponse systems. We will call the password presenter the prover and the presentee the 
verifier. The methods we present can be used for two-way authentication. That is the 
same method can both authenticate the prover to the verifier and authenticate the verifer to 
the prover. To each prover in conjunction with a standard password there will be assigned a 
finitely presented group with a solvable word problem. This is the challenge group. This 
will be done randomly by the group randomizer system and will be held in secret by the 
prover and the verifier. Cryptographically we assume the adversary can steal the encrypted 
form of the group theoretic responses. Probabilistically this does not present a problem. 
Each challenge response set of questions forms a virtual one time key pad. Therefore the 
adversary must steal three things - the original password, the challenge group and the group 
randomizer. Hence there is almost total security in the challenge response system. Further 
there is an infinite supply of finitely presented groups to use as challenge groups and an 
infinite supply of challenge response questions that never have to be duplicated. 

The theoretical security of the system is provided by several results in asymptotic group 
theory. In particular a result of Lysenok [L] implies that stealing the challenge group is NP- 
hard while a result of Jitsukawa [J] says that the asymptotic density of using homomorphisms 
to attack the group randomizer protocol is zero (see sections 9 and 10) 
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In brief outline the system works as follows. Wc assume that each prover has a group 
randomizer system. At the most basic level the group randomizer system has the ability to 
do the following things: 

(1) Recognize a finite presentation of a finitely presented group with a solvable word prob- 
lem and manipulate arbitrary words in the alphabet of generators according to the rewriting 
rules of the presentation. In particular if the group is automatic the group randomizer can 
rewrite an arbitrary word in the generators in terms of its group normal form. 

(2) Given a finite presentation of a group with a solvable word problem recognize whether 
two free group words have the same value in the given group when considered in terms of 
the given generators of the group 

(3) Randomly generate free group words on an alphabet of any finite size 

(4) Recognize and store sets of free group words Wi, Wk on an alphabet x\,...,Xn and 
rewrite words W(H^i, Wk) as the corresponding word in Xx, a;„. 

(5) Given a free group of finite rank on Xi, a;„ and a set of words Wi, Wk on an 
alphabet xi,..,Xn solve the membership problem in F relative to H —< Wi, Wk >, the 
subgroup of F generated by Wi,..., Wk- 

(6) Given a stored finitely presented group or a stored set of free group words the 
randomizer can accept a random free group word and rewrite it as a normal form in the 
finitely presented group in the former case or as a word in the ambient free group in the 
latter case. 

Each prover further has a standard password. Suppose that _F is a free group on {xi, x„}. 
The prover's password is linked to a finitely generated subgroup of a free group given as 
words in the generators - that is the prover's password is linked to Wi, Wk where each 
Wi is a word in xi,...,x„. The group G =< Wi,...,Wk > is called the challenge group. 
In general k ^ n. The prover doesn't need to know the generators. The randomizer can 
randomly choose words from this subgroup and then freely reduce them. The verifier has 
the challenge group or subgroup also stored in its randomizer. From the viewpoint of 
cryptology this is a symmetric key protocol with both prover and verifier having a common 
shared secret, {P,G), where P is a standard password and G is the challenge group. The 
shared secret is set at initialization of the protocol by some direct communication. This is 
the most common model for password security. 

The prover submits his or her standard password to the verifier. This activates the verifier's 
randomizer to the prover's set of words. The verifier now submits a random free group 
word on yi,...,yk to the prover's randomizer say yfe). The prover's randomizer 

treats this as W{Wi, ...,Wk) and then reduces it in terms of the free group generators 
Xi, ...,Xn and rewrites it as W*{xi, ...,x„). The verifier checks that this is correct - that is 
W{Wi, Wk) = W*{xi, x„) on the free group on xi, x„. If it is the verifier continues 
and does this three (or some other finite number) of times. There is one proviso. The verifier 
submits a word to the prover only once so that a submitted word can never be reused. The 
prover's randomizer will recognize if it has (this is a verification to the prover of the verifier). 

To verify that the verifier is legitimate the process is repeated from the prover's randomizer 
to the verifier. 
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In [BBFT] several variations of this basic outline using more general finitely presented groups 
were presented. Using some results from asymptotic group theory it was shown that this 
method is provably secure. 

7 Braid Group Cryptography and Platform Groups 

As platform groups for their respective protocols, both Ko-Lee and Anshel-Anshel-Goldfeld 
suggested the braid groups i?„ (see [Bi]). The groups in this class of groups possess the 
desired properties for the key exchange and key transport protocols; they have nice presen- 
tations with solvable word problems and conjugacy problems; the solution to the conjugacy 
and conjugator search problem is "hard"; there are several possibilities for normal forms 
for element and they have many choices for large commuting subgroups. Initially the braid 
groups were considered so ideal as platforms that many other cryptographic applications 
were framed within the braid group setting. These included authentication and digital 
signatures. There was so much enthusiasm about using these groups that the whole area of 
study was named braid group cryptography. A comprehensive and well-written article 
by Dehornoy [D] provides a detailed overview of the subject and we refer the reader to that 
for technical details. 

After the initial successes with braid group cryptographic schemes there were some sur- 
prisingly effective attacks. There were essentially three types of attacks; an attack using 
solutions to the conjugacy and conjugator search problems, an attack using heuristic proba- 
bility within Bn and an attack based on the fact that there are faithful linear representations 
of each i?„ (see [D]). What is most surprising is that the Anshel-Anshel-Goldfeld method 
was susceptible to a length based attack. In the Anshel-Anshel-Goldfeld method the pa- 
rameters are the specific braid group Bn and the rank of the secret subgroups for Bob 
and Alice. A length based attack essentially broke the method for the initial parameters 
suggested by Anshel, Anshel and Goldfeld in [AAG] . The parameters were then made larger 
and attacks by this method were less successful. However this led to research on why these 
attacks on the conjugator search problem within Bn were successful, What was discovered 
was that generically (sec sections 9 and 10) a random subgroup of Bn is a free group and 
hence length based attacks are essentially attacks on free group cryptography and therefore 
STiccessful. What this indicated was that althoiigh randomness is important in cryptography 
in using the braid groups as platforms subgroups cannot be chosen purely randomly. 

Braid groups arise in several diflFerent areas of mathematics and have several equivalent 
formulations. What we do in the remainder of this section is describe the braid groups. A 
complete topological and algebraic description can be found in the book of Joan Birman 
[Bi]. 

A braid on n strings is obtained by starting with n parallel strings and intertwining them. 
We number the strings at each vertical position and keep track of where each individual 
string begins and ends. We say that two braids are equivalent if it is possible to move the 
strings of one of the braids in space without moving the endpoints or moving through a 
string and obtain the other braid. A braid with no crossings is called a trivial braid. We 
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form a product of braids in the following manner. If u is the first braid and v is the second 
braid then uv is the braid formed by placing the starting points for the strings in v at the 
endpoints of the strings in u. The inverse of a braid is the mirror image in the horizontal 
plane. It is clear that if we form the product of a braid and its mirror image wc get a braid 
equivalent to the trivial braid. With these definitions, the set of all equivalence classes of 
braids on n strings forms a group B„. We let cr, denote the braid that has a single crossing 
from string i over string i + 1. Since a general braid is just a series of crossings it follows 
that Bn is generated by the set (Ti;i = l,...,n — 1. 

There is an equivalent algebraic formulation of the braid group iJ„ . Let F„ be a free group 
on the n generators xi, Xn with n > 2. Let crj, i = 1, n — 1 be the automorphism of Fn 
given by 

Ci '. Xi I ^ Xi-\^i^Xi^i I ^ X^^^XiXi-\-i 

(Ji;Xj 1-^ Xj,j ^ i,i + I. 

Then each <Ji corresponds precisely to the basic crossings in B^- Therefore i?„ can be 
considered as the subgroup of Aut{Fn) generated by the automorphisms cr,, Artin proved 
[A] (see also [MKS]) that a finite presentation for S„ is given by 

B„ =< (71, ...,cr„_i; [ai,aj] = 1 if \i - j\ > l,Xi+iXiXi+i = XiXi+iXi,i = 1, ...,n- 1 > . 

This is now called the Artin presentation. The fact that B„ is contained in Aut{Fn) provides 
an elementary solution to the word problem in _B„ since one can determine easily if an 
automorphism of Fn is trivial on all the generators. We note that although the braid groups 
Bn are linear ( the Lawrence-Krammer representation is faithful (see[D]) it is known that 
Aut{Fn) is not linear (see [F]). 

There arc several possibilities for normal forms for elements of B„. The two most commonly 
used are the Garside normal form and the Dehornoy normal form. These are described 
in [D] and [MSU]. 

From the commuting relations in the Artin presentation it is clear that each Bn has the 
requisite collection of commuting subgroups. 

The conjugacy problem for _B„ was originally solved by Garside and it was assumed that 
it was hard in the complexity sense. Recently there has been significant research on the 
complexity of the solution to the conjugacy problem (see [MSU] and [D]). 

In general, platform groups for the non-commutative protocols that we have discussed re- 
quire certain properties. Most are present in the braid groups. The first is the existence 
of a normal form for elements in the group. Normal forms provide an effective method of 
disguising elements. Without this, one can determine a secret key simply by inspection 
of group elements. The existence of a normal form in a group implies that the group has 
solvable word problem, which is essential for these protocols. For purposes of practicality, 
the group also needs an efficiently computable normal form, which ensures an efficiently 
solvable word problem. 

In addition to the platform group having normal form, ideally, it would also exhibit ex- 
ponential growth. That is, the growth function for G, 7 : N R defined by 7(71) = 
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4j= {w G G : l{w) < n}, has an exponential growth rate. Exponential growth is a necessity 
since this ensures that the group will provide a large key space, making a brute force search 
for the secret key an infeasible algorithm. 

The other property which is necessary in most of the proposed cryptosystems is the conjuga- 
tor search problem for the platform group ideally should have exponential time complexity. 

In addition to these, for the Ko-Lee type protocols, we need large commuting subgroups 
wihtin the platform group. 

Currently, there are many potential platform groups that have been suggested. The following 
are some of the proposed platform groups: 

• Braid groups(Ko-Lee, Anshel-Anshel-Goldfeld) 

• Thompson Groups (Shpilrain-Ushakov) [SU] 

• Polycyclic Groups (Eick-Kahrobaci) [EK] 

• Linear Groups (Baumslag-Finc-Xu) [BEX 1,2] 

• Free metabelian Groups (Shpilrain-Zapata) [SZ] 

• Grigorchuk Groups (Petrides) [P] 

• Groups of Matrices (Grigoriev-Ponomarenko) [GP] 

• Surface Braid Groups (Camps) [C] 

Many of these are discussed in [MSU]. 

8 Cryptography With Polycyclic Groups 

In this section we briefly discuss polycyclic group cryptography which has not been exten- 
sively studied but has many of the essential features for ideal platform groups. A cryptosys- 
tem using polycyclic groups was developed in [EK]. 

A group G is called polycyclic if it has a series G = Gn+i > G„ > • ■ • > G2 > Gi = 1 
in which each Gi is a normal subgroup of Gj+i and Gi+i/Gi is cyclic for i = 1, 2, • • • , n. 
A series of this type is called a polycyclic series. Polycyclic groups are a natural non- 
commutative generalization of cyclic groups. The book of Holt et,al. [HEO] is a good 
reference for information about polycyclic groups. 

Every polycyclic group G has a finite presentation of the form: 



for 1 < j < i < n where € N U 00, < 00 if i G / C {1,2,-- - ,n} and Wij,Vij,Ujj 
are words in the generators a^+i, • • • , an- If rj = [Gj+i : G,] for each i G {1, 2, • • • , n} 
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then this presentation is called a consistent polycyclic presentation. Every element in the 
group defined by this consistent polycyclic presentation may be written uniquely in the form 
O'T ■ ■ ■ o^r with Cj G Z and < ej < if t € /. This unique representation of each element 
.g G G is called the normal form of G. It is known that every polycyclic group exhibits a 
consistent polycyclic presentation. Hence, every polycyclic group has a normal form. This 
is used as a basis for computations with polycyclic groups. 

The word problem can be solved effectively using the collection algorithm in a group G 

given by a consistent polycyclic presentation. The collection algorithm compiites the unique 
normal form for an element, g, in the group given by a word in the generators. This is 
done by repeatedly applying the power and conjugacy relations given in the presentation to 
subwords of 17, transforming g to an equivalent word. The nature of the relations ensures 
that this process must terminate, producing the unique normal form for g. The collection 
algorithm is known to be a practical and effective method for solving the word problem in 
consistent polycyclic presentations. 

Every polycyclic group can be embedded in GL{n,Z), which reveals important properties 
about polycyclic groups. Since matrix multiplication is solvable in polynomial time, group 
multiplication in polycyclic groups is efficient. As polycyclic groups have a normal form, 
efficiently solvable group multiplication implies that the word problem is also efficiently 
solvable. It has been proven that the search conjugacy problem in any subgroup of a general 
linear group is solvable. Because every polycyclic group can be embedded as a subgroup of 
GL(n, Z), the search conjugacy problem in polycyclic groups is solvable. The complexity of 
the search conjugacy problem in polycyclic groups is unknown, but widely conjectured to 
be exponential time. 

Recall that the Anshcl-Anshel-Goldfcld key exchange can be broken if the simultaneous 
search conjugacy problem is solvable. In polycyclic groups, the simultaneous search conju- 
gacy problem reduces to the search conjugacy problem. 

9 Generic Complexity and Asymptotic Density 

In cryptanalysis involving group theoretic decision problems what is important is not just 
the solution to the problem but the computational complexity, polynomial or exponential 
for example, of the algorithm to solve the problem. The problem may be hard or even 
undecidable on some inputs but actually easy on most inputs. This is in reality a problem 
in many braid group schemes. The conjugator search problem is hard on some inputs but 
actually easy for many chosen subgroups. A problem may be very hard on some inputs. 
This is called worst case complexity. More important for cryptanalysis is average case 
complexity, that is the complexity on average over all inputs. Generic complexity refers 
to the complexity of the solution to a particular algorithm over most inputs. Generic and 
average case complexity and their uses in cryptanalysis are discussed in detail in [MSU]. 
There they show that generic complexity is a more useful tool in most cryptographic ap- 
plications. There they show that in many cases if an algorithm is easy on average it is 
also easy generically. They also show that the opposite is not true and provide examples of 
algorithms that are exponential on average but polynomial time generically. 
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The problem with some braid group eryptographie algorithms is that random subgroups of 
Bn are gcnerically free. We now deseribe what this means. 

Asymptotic density is a general method to compute densities and/or probabilities on infinite 
discrete sets where each individual outcome is tacitly assumed to be equally likely. The 
method can also be used where some probability distribution is assumed on the elements. It 
has been effectively applied to determining densities within infinite discrete finitely generated 
groups where random elements are considered as being generated from random walks on the 
Cayley graph of the group. The paper by Borovik, Myasnikov and Shpilrain [BMS] provides 
a good general description of this method in group theory. Let P he a group property and 
let G be a finitely generated group. We want to determine the measure of the set of elements 
which satisfy V. For each positive integer n let _B„ denote the n-ball in G. Let |_B„| denote 
the actual size of Bn (which is an integer since G is finitely generated) or the measure of 
\Bn\ if a distribution has been placed on the elements of G. Let S be the set of elements in 
G satisfying V. The asymptotic density of S is then 

\snBn\ 

lim — — — — 

provided this limit exists. We say that the property V is generic in G if the asymptotic 
density of the set S of elements satisfying V is one, V is called an asymptotic visible prop- 
erty, if the corresponding asymptotic density is strictly between and 1. If the asymptotic 
density is 0, then V is called negligible. 

This concept can be easily extended to properties of finitely generated subgroups, We con- 
sider the asymptotic density of finite sets of elements that generate subgroups that have a 
considered property. For example to say that a group has the generic free group property 
we mean that 

\S„, n B„i,,i] 

hm — j = 1 

m,n^cx> \Bm,n\ 

where Sm is the collection of finite sets of elements of size m that generate a free subgroup 

and Bm,n is the collection of m clement subsets within the n-ball. 

If P is a group property and G is a group then we say that subgroups of G are generically 
P if a generic randomly chosen subgroup H of G has property V. Equivalently this means 
that the asymptotic density of subgroups H oi G that have property V is one. 

10 The Generic Free Group Property 

In general we say that a group G has the generic free group property if a finitely 
generated subgroup is generically a free group. A result of Epstein [E] shows that the group 
GL{n, M.) satisfies the generic free group property. Further G has the strong generic free 
group property if given randomly chosen elements gi,...,gn in G then generically they 
are a free basis for the free subgroup they generate. Jitsukawa [J] showed that finitely 
generated nonabelian free groups have the strong generic free group property while Oilman, 
Myasnikov and Osin [GMO] showed that torsion-free hyperbolic groups also have the generic 
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free group property. Myasnikov and Ushakov [MU] showed that pure braid groups P„ with 
n > 3 also have the strong generic free group property. A recent result of Carstensen, Fine 
and Rosenberger [CFR] shows that all Fuchsian groups of finite co-volume and all braid 
groups B„with n > 3 have the strong generic free group property, 

This result of Myasnikov and Ushakov on the pure braid groups has applications to the 
cryptanalysis of both the Ko-Lee cryptosystem and the Anshel-Anshel-Goldfeld cryptosys- 
tem (see [SU] and [MU]). Both cryptosystems were usually suceptible to length based attacks 
if the parameters chosen in the braid groups i?„ were small. The reason for this is that ran- 
dom choices of subgroups within the braid groups are actually free groups. This does not 
disqualify the braid groups as platforms but rather says that subgroups cannot be chosen 
entirely randomly. 

Extremely useful in proving that a group has the generic or strong generic free group prop- 
erty is the following. 

Theorem 10.1. Let G be a group and N a normal subgroup. If the quotient G/N satisfies 
the strong generic free group property then G also satisfies the strong generic free group 
property. 

In [FMR] it was shown that many group amalgams - free products, free products with 
amalgamtion and HNN groups, satisfy the strong generic free group property. In particular 
the most general result is the following. 

Theorem 10.2. Let A and B be arbitrary finitely generated infinite groups and let G = 

A-k B be their free product. Let {xi, x„} be n randomly chosen elcm,cnts from G. Then 
generically these elements are a free basis for the subgroup they generate, that is G satisfies 
the strong generic free group property. 

This can bo extended to more general amalgams in many ways (see [FMR]) 

Theorem 10.3. Let A and B be arbitrary finitely generated infinite groups and let G — A^B 

be their amalgamated free product with amalgamated subgroup H. Let H\ and H2 be the copy 

of H in A and B respectively. Suppose that A/N{Hi) is infinite and B/N{H2) is infinite 
where N{Hi) is the normal closure of Hi in the respective factors. Then G satisfies the 
strong generic subgroup property. 

Recall that a cyclically pinched one-relator group is a amalgamated free product of 
the form 

G = Fi ★ F2 

{U=V} 

where Fi , F2 are finitely generated free groups and U, V arc nontrivial words in the respective 
free groups. If U is not a power of a primitive element in Fi and V is not a power of a 
primitive element in F2 then the quotient of Fi and F2 by the normal closure of U and V 
respectively is a nontrivial, infinite one-relator group. 

Corollary 10.1. Let G be a cyclically pinched one-relator group as above. Assume that U 
and V are not a power of a primitive element in Fi and F2 respectively. Then G satisfies 
the strong generic subgroup property. 
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In particular any oricntable surface group of genus g > 2 falls into the class of cyclically 

pinched onc-rclator groups. 

Corollary 10.2. Any orientable surface group of genus g > 2 and any nonorientable surface 
group of genus g > 4 satisfies the strong genciic subgroup property. 

The situation with HNN groups becomes even more complicated but some things can be 
proved as consequences of the amalgam result above. Notice first however that any HNN 
group with free part of rank > 2 must have a free quotient of rank > 2 and hence satisfy 
the strong generic subgroup property. Therefore only the case where the free part has rank 
1 must be considered. 

Theorem 10.4. Let G be an HNN extension of the group B with a presentation 

G =< t, B; rel{B), t-^Ut = V > 

with U,V nontrivial isomorphic subgroups of B. Let Nb{< U,V >) be the normal closure 
of the subgroup < U,V > in B. Then if B/Nb{< U,V >) is infinite, G satisfies the strong 
generic subgroup property. 

Extensions of centralizers play a large role in the study of the elementary theory of free 
groups. Recall that if B is a group and U £ B then a rank one extension of centralizers 
of B is a group with a presentation 

G=< t,B;rel{B),t-^UT = U > . 

Theorem 10.5. Let G be a rank one extension of centralizers of the group B. Suppose G 
has a presentation 

G =< t,B;rel{B),t-^Ut = U> 

wihere U is a nontrivial element of B. If B/Nb{U) is infinite, where Nb{U) is the normal 
closure of U in B, then G satisfies the strong generic subgroup property . 

In the situation where the factors are finite we must be careful even for free products. The 
infinite dihedral group Z2 ★ Z2 is solvable so cannot satisfy the strong generic free group 
property. However if at least one factor has order greater than 2, an analysis based on 
Kurosh bases yields the weaker generic free group property. 

Theorem 10.6. Let G = AirB be a nontrivial free product. If at least one factor has order 
greater than 2 then G satisfies the generic free group property. 

In [CFR] it was shown that a finitley generated group satisfies the strong generic free group 
property if and only if subgroups of fintie index do also. We call a group property V suitable 
for a finitely generated group G if it is preserved under isomorphisms and its asymptotic 
density is independent of finite generating systems. From a result in [MSU] the strong 
generic free group property is suitable in any group that has a nonabelian free quotient. 

The main result in [CFR] is the following called the inheritance theorem. 
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Theorem 10.7. Let G be a finitely generated group and H < G a subgroup of finite index 
[G : H] = n < 00. Let V be the strong generic free group property. Then: 

1. IfV is a suitable and generic property in H then it is also suitable and generic in G. 

2. IfV is a suitable and generic property in G then it is also suitable and generic in H. 

An interesting consequence of this is that all braid groups have the strong generic free group 
property. 

11 Open Problems 

Wo now give a nonexhaustive list of problems related to the rest of this article. 
General: 

1. What is the most appropriate platform group for non-commutative cryptography? 

2. Should the group the be finite or infinite? 

3. How can we show a group is provably secure for the new non-commutative schemes 
such as public key exchanges, digital signatures and authentication? 

4. Can wc design more public keys based on other search and decision problems in com- 
binatorial group theory? 

5. Can we analyze the security of this protocols? 

6. what should be the measure of the security? (practicality, complexity, average case 

complexity, generic complexity?) 

7. So far there are three known non-commutative digital signatures have been designed, 
can we design more non-commutative digital signatures? 

8. What about the authentication schemes? 

9. What is the appropriate choice of commuting subgroups in poly cyclic groups which 
makes the described schemes secure? 

Complexity Analysis and Security: 

1. What is the complexity of the search conjugacy problem in polycyclic groups? 

2. What is the complexity of the decision conjugacy problem, for example in Eick- 

Ostheimer algorithm [EO]? 

3. What is the complexity of the collection algorithm to find the normal form of elements 
in polycyclic groups? 
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4. What is the complexity of the twisted search conjugacy problem in polycyclic groups? 

5. What is the complexity of the Reidemeister-Schreier rewriting algorithm in free groups? 

6. What is the generic case complexity of the search conjugacy problem in polycyclic 
groups? 

Qucintum Algorithms and Quantum Complexity: 

Another problem to think in this direction is quantum computational approaches to these 
cryptosystems. Quantum algorithms for finite solvable groups (which are polycyclic) has 
been studied, particularly by J. Watrous (2001) [Wa]. He found a quantum algorithm to 
compute the order of a finite solvable group in polynomial time. Algorithm works in the 
setting of black-box groups none of them have polynomial-time classical algorithms. Can we 
design quantum algorithms for solving other decision problems in polycyclic groups (both 
for finite and infinite ones); especially the ones we use in cryptography. 

1. Is there any quantum algorithm for solving the search conjugacy problem for polycyclic 
group that reduces the complexity of the algorithm? 

Implementation: 

1. How can we implement the proposed cryptosystems? 

2. Can the computer algebra system GAP, could we use this for practical and secure 
cryptography? 
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